Aggregation of judgments

Aggregation of judgments is the process by which separate, often local opinions become a collective decision or estimate. It is the indispensable counterpart to decentralization in the book’s account of collective intelligence: diversity and independence put information into the system, but aggregation makes that information usable as a whole. Without it, a group’s best available result may be only the solution produced by its smartest member; with it, the group can sometimes do better than any individual within it.

Function and Conditions

The concept appears within a four-part account of when a crowd can make a good judgment. Its members need diversity of opinion, independence from one another’s judgments, decentralization that lets them specialize and draw on local knowledge, and an aggregation mechanism that turns their separate assessments into a collective decision. The first three conditions create and preserve useful differences; aggregation gathers those differences into a result. Remove the final condition and the crowd may possess a wealth of knowledge without being able to act on it.

Aggregation also has a statistical rationale. A person’s estimate contains both information and error. When a sufficiently large group of diverse, independent people makes a prediction or estimates a probability, averaging their judgments can cancel out some individual errors and leave a result that reflects the information held among them. This does not guarantee a correct answer: error can cancel while the group still lacks relevant information. The force of the method depends both on the quality of what participants know and on a process that can combine their contributions.

The January-day reaction to the Challenger explosion offers a case in which the mechanism is a market price. Traders did not need to share one person’s complete account of what had happened to the O-rings or to know individually that Thiokol was responsible. Each brought whatever information or interpretation was available to them; collectively, the market’s judgment reflected that Thiokol was responsible. The account presents the episode not as evidence that every trader possessed inside knowledge, but as an instance of dispersed pieces of information combining into a conclusion that individuals might not have reached with certainty on their own.

Mechanisms in Practice

The mechanism changes with the setting. In the experiment with which the book began, Francis Galton aggregated judgments by counting the votes. In a free market, price performs that role: it imperfectly but effectively reflects buyers’ and sellers’ actions and conveys investors’ judgments about a company’s worth. In a sports betting market, the final point spread similarly records bettors’ collective judgment of a game’s likely outcome. The line may begin with a bookmaker, but it shifts in response to bets; at the Mirage, an early pair of $3,000 wagers on Baltimore was enough to prompt a change in the spread before the 2001 Super Bowl.

The betting example also clarifies what aggregation can and cannot promise. Point spreads only weakly predict the final scores of most NFL games, yet gamblers find it difficult to beat the final spread consistently. Favorites cover in roughly half of games and underdogs in the other half, and the book’s account reports no obvious, persistent error such as home teams regularly winning by more than the crowd expects. The Mirage’s final line is described as the most reliable available forecast about three-quarters of the time. Aggregated judgment is not prophecy; its strength is that the combined estimate is hard to outperform consistently.

Election markets offer another mechanism. In the Iowa Electronic Markets, one kind of contract pays according to a candidate’s share of the popular vote: a George W. Bush contract bought for the 2000 election would have paid 48 cents because he received 48 percent of the vote. Across forty-nine elections from 1988 to 2000, election-eve prices were, on average, off by 1.37 percent in presidential elections, 3.43 percent in other U.S. elections, and 2.12 percent in foreign elections. The market generally outperformed major national polls, and, among 596 polls released during presidential elections in that period, its price on the day each poll appeared was more accurate three-quarters of the time. Its prices also tended to be less volatile than poll results, changing sharply mainly when new information arrived.

The market had never more than about 800 traders and did not mirror the electorate: most traders were men, and a disproportionate, though shrinking, share came from Iowa. They were not simply predicting their own votes. Yet their prices forecast national voting behavior better than asking voters what they intended to do. Here aggregation did not require a miniature version of the population being forecast; it relied on a market in which participants’ judgments could be expressed and combined.

Decentralization Needs a Common Result

The relationship between aggregation and decentralization is deliberately paradoxical. Decentralization permits people to work independently, specialize, and use knowledge specific to their own experience. But information discovered in one corner of a system may never reach the rest. A large group can therefore be rich in local knowledge and poor in collective understanding. The challenge is to make private and local information globally useful without erasing the conditions that allowed people to acquire it.

Prices show one way to meet that challenge: they communicate the combined actions of buyers and sellers while leaving decisions dispersed. The book also invokes Google’s use of the local knowledge of millions of Web-page operators to make searches smarter and quicker. Linux illustrates a different arrangement. Programmers are distributed around the world, but potential changes to the operating-system source code are vetted by a small number of coders, including Linus Torvalds. The author questions whether such a small group needs to control the decision; a larger group of programmers, even if less skilled than Torvalds and his lieutenants, might evaluate code well. Still, some means of deciding which contributions to keep is necessary. Without any means of combining judgments, autonomous contributors can do no better than the solution produced by the group’s smartest individual, and there is no guarantee that the group will choose that solution.

Aggregation is therefore described as a curious form of centralization, but not as central planning. Collective decisions can emerge from decentralized agents, and an organizing mechanism need not concentrate all judgment in a small command structure. The distinction matters because a decentralized system without aggregation may be disorganized, while one that gathers judgments can use distributed knowledge without making every decision at the center.

Failure When Judgments Do Not Travel

The pre-September 11 U.S. intelligence community shows what happens when information is decentralized but not aggregated. Dividing work among agencies could encourage specialization and give analysts different perspectives on the same problem. But, as the book frames the failure, the system lacked a genuine means of combining not only information but judgments. National Security Agency analysts, CIA officers, and FBI agents could each hold relevant knowledge without any mechanism that allowed the intelligence community to assess its collective significance.

Richard Shelby’s proposed remedy was a truly central intelligence agency, which could make one agency responsible for the information. That might address the sharing problem, but it would also risk sacrificing diversity, local knowledge, and independence—the advantages that specialization and decentralization provide. The argument is not that information should remain hidden within agencies. It is that sharing it does not require a small central group to do all the sifting. The book argues instead for aggregation: connect the judgments of the larger community rather than assume that a few people at the center will be smarter than the whole.

Some possible steps were mechanical. Linking agency databases could improve information flow while allowing agencies to retain autonomy. Yet two years after September 11, the government still lacked a single unified watch list drawing on data from across the intelligence community. A more ambitious proposal, FutureMAP, would have created decision markets modeled on the Iowa Electronic Markets. Intelligence analysts and perhaps outside experts could buy and sell futures contracts based on expectations about events in the Middle East and elsewhere. The initiative received funding from the Defense Advanced Research Projects Agency but was aborted. Together, the examples distinguish gathering information from aggregating judgments: the first makes material available; the second supplies a way for dispersed assessments to contribute to a collective forecast.

Scientific and Organizational Forms

The collaborative search for the cause of SARS offers a decentralized scientific form of aggregation. The World Health Organization contacted eleven laboratories in countries around the world on March 15 and 16, 2003, and the laboratories began a joint research project on March 17. They held daily teleconferences, posted electron-microscope photographs, analyses, and test results to a WHO website, and traded virus samples so that they could check and learn from one another. Different laboratories could investigate the same samples at the same time. By March 21, researchers at Hong Kong University had isolated a likely candidate, while scientists at the Centers for Disease Control in the United States separately isolated a coronavirus. Laboratories then detected the coronavirus across samples, sequenced it, and tested its effects; by April 16, only a month after the collaboration began, they were confident it caused SARS.

No single scientist’s contribution alone constituted the discovery. Cynthia Goldsmith, an electron microscopist at the Centers for Disease Control and Prevention laboratory in Atlanta, first spotted the coronavirus, but weeks of work by laboratories around the world were needed to show that it caused the disease. The work that ruled out other viruses also narrowed the field. The World Health Organization orchestrated the network, but did not dictate which samples or viruses each laboratory should investigate or how it should exchange information. The laboratories agreed to share relevant data and speak every morning; beyond that, they worked out how to divide the work. The discovery was consequently attributed to the laboratories collectively, not to one individual. Their arrangement gathered independent investigation without placing a director in charge of every analytic judgment.

Scientific publication extends this process beyond formal collaborations. Scientists generally publish findings and make data available for inspection, allowing others to reconsider results, challenge conclusions, and use the information to form hypotheses and experiments of their own. Even failed hypotheses contribute by showing colleagues where not to look. Scientific knowledge is thus not only cumulative, inherited from earlier researchers, but collective: contemporary researchers rely on one another’s work, and their community evaluates whether a claimed discovery is accepted into common knowledge.

That communal acceptance does not make truth a matter of opinion. The coronavirus caused SARS before the World Health Organization announced it. But in scientific practice, the claim became an established contribution only after other scientists scrutinized the work and accepted it as demonstrating that causal relationship. Subsequent work on tests and vaccines proceeded on that basis. This is an indirect form of consensus, without literal voting or market prices, but it too aggregates judgments: discoveries become part of the common fund of knowledge through the community’s assessment rather than through the pronouncement of an elite few.

Limits and Consequences

Aggregation cannot compensate for a process that suppresses independent information before it can be combined. NASA’s Mission Management Team discussion of the debris strike illustrates the contrast. The team’s discussion was structured around a question from Linda Ham and an answer from someone else, but it did not ensure that members commented on specific issues or that minority views received a hearing. The team began with its conclusion and reinterpreted incoming information to fit it. The engineers on the Debris Assessment Team had serious concerns and wanted images of the Orbiter’s wing before reaching a fully informed analysis, yet those concerns never received a serious hearing. With dissent and relevant analysis excluded, there was little for the decision-making process to aggregate.

A similar blockage can occur inside a corporation. In one company, a new product idea had to navigate 223 different paths through the bureaucracy before it could be accepted. More than fifteen management layers separated Ford’s chairman from a factory-floor supervisor, compared with five at Toyota. Such distance made it harder for information from workers and local settings to reach decision-makers, while multiple layers could become obstacles to its passage. Aggregation depends not merely on having many people or collecting their reports, but on channels that let knowledge move and on procedures that make room for differing judgments.

The organizational alternative is not to replace every manager with a crowd. Decisions about local problems can often be made by people close to those problems, whose experience gives them tacit knowledge that is difficult to summarize or transmit upward. But specialization and local authority must be balanced with ways to make the resulting knowledge useful across the organization. Aggregation is the connecting principle: it allows decentralized contributors to retain their distinct knowledge while giving the larger group a means to form a collective judgment. The book’s examples—from markets and elections to intelligence, science, software, and firms—make the same point in different forms: dispersed information becomes collective intelligence only when a working mechanism brings judgments together.